Methods and systems for distinguishing objects in a natural setting to create an individually-manipulable volumetric model of an object

ABSTRACT

An exemplary virtual reality media provider system receives two-dimensional (“2D”) video data for surfaces of first and second objects located in a natural setting. The 2D video data is captured by first and second capture devices disposed at different positions with respect to the objects. The system distinguishes the first object from the second object by performing a plurality of techniques in combination with one another. The plurality of techniques include determining that the first object is moving in relation to the second object; and determining that, from a vantage point of at least one of the different positions, a representation of the first object captured within the 2D video data does not overlap with a representation of the second object. Based on the received 2D video data and the distinguishing of the first and second objects, the system generates an individually-manipulable volumetric model of the first object.

RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 15/141,717, filed Apr. 28, 2016, and entitled“Methods and Systems for Creating and Manipulating anIndividually-Manipulable Volumetric Model of an Object,” which is herebyincorporated by reference in its entirety.

BACKGROUND INFORMATION

Advances in computing and networking technology have made new forms ofmedia content possible. For example, virtual reality media content isavailable that may immerse viewers (or “users”) into interactive virtualreality worlds that the users may experience by directing theirattention to any of a variety of things being presented in the immersivevirtual reality world at the same time. For example, at any time duringthe presentation of the virtual reality media content, a userexperiencing the virtual reality media content may look around theimmersive virtual reality world in any direction with respect to both ahorizontal dimension (e.g., forward, backward, left, right, etc.) aswell as a vertical dimension (e.g., up, down, etc.), giving the user asense that he or she is actually present in and experiencing theimmersive virtual reality world from a particular viewpoint (e.g.,vantage point) within the immersive virtual reality world.

In some examples, a virtual reality media provider may provide virtualreality media content that includes an immersive virtual reality worldrepresentative of real-world objects and scenery (i.e., as opposed tocomputer-generated, animated, or other virtual objects and scenery). Forexample, the immersive virtual reality world may represent a real-worldevent (e.g., a sporting event, a concert, etc.) that may be taking placein real time (e.g., a live event), a fiction or non-fiction live-actionprogram (e.g., a virtual reality television show, movie, documentary,etc.), or another type of program involving real-world objects andscenery.

Traditionally, immersive virtual reality worlds based on real-worldobjects and scenery are “flat” in the sense that all the real-worldobjects and/or scenery of the immersive virtual reality world arerepresented within virtual reality media content in the aggregate, as aflat conglomerate scene as viewed by a person standing at a fixed spotin the real world. This may be true even in cases where stereoscopicallydifferent versions of the flat conglomerate scene may be presented toeach eye of a user experiencing the immersive virtual reality world togive the scene a three-dimensional appearance. As a result, theimmersive virtual reality world may look realistic to the user from theparticular viewpoint of the user, but there may be significantlimitations placed on specific objects included in the flat conglomeratescene as to how the specific objects may be presented, manipulated,modified, and so forth with respect to other objects included in theflat conglomerate scene and with respect to the immersive virtualreality world in general.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate various embodiments and are a partof the specification. The illustrated embodiments are merely examplesand do not limit the scope of the disclosure. Throughout the drawings,identical or similar reference numbers designate identical or similarelements.

FIG. 1 illustrates an exemplary virtual reality media provider systemthat may create and manipulate an individually-manipulable volumetricmodel of an object located in a natural setting according to principlesdescribed herein.

FIG. 2 illustrates an exemplary implementation of the virtual realitymedia provider system of FIG. 1 according to principles describedherein.

FIG. 3 illustrates an exemplary configuration in which the virtualreality media provider system of FIG. 1 operates to create andmanipulate an individually-manipulable volumetric model of an objectlocated in a natural setting according to principles described herein.

FIG. 4 illustrates exemplary media player devices configured tofacilitate a user in experiencing an immersive virtual reality worldthat includes virtual reality media content withindividually-manipulable volumetric models of objects according toprinciples described herein.

FIG. 5 illustrates an exemplary technique for creating anindividually-manipulable volumetric model of an exemplary object in anatural setting according to principles described herein.

FIG. 6 illustrates exemplary techniques for distinguishing a firstobject located in a natural setting from a second object located in thenatural setting along with the first object according to principlesdescribed herein.

FIG. 7 illustrates an exemplary dataflow for creating and manipulatingan individually-manipulable volumetric model of an object located in anatural setting according to principles described herein.

FIG. 8 illustrates an exemplary virtual reality experience in which auser is presented with exemplary virtual reality media contentrepresentative of an immersive virtual reality world corresponding to anexemplary natural setting containing an object for which anindividually-manipulable volumetric model has been generated accordingto principles described herein.

FIG. 9 illustrates exemplary manipulations that may be performed on theindividually-manipulable volumetric model of FIG. 8 with respect to theimmersive virtual reality world of FIG. 8 while a user is experiencingthe immersive virtual reality world according to principles describedherein.

FIGS. 10 and 11 illustrate exemplary methods for creating andmanipulating an individually-manipulable volumetric model of an objectlocated in a natural setting according to principles described herein.

FIG. 12 illustrates an exemplary computing device according toprinciples described herein.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Methods and systems for creating and manipulating anindividually-manipulable volumetric model of an object (e.g., areal-world object) are described herein. Individually-manipulablevolumetric models of objects may be useful for many purposes,particularly when the individually-manipulable volumetric models may becreated from objects located in a natural setting (e.g., as opposed to astudio setting with a “green screen”). For example, virtual realitymedia content that is based on individually-manipulable volumetricmodels of real-world objects rather than on a flat conglomerate scenerepresentative of objects and scenery as they appear from one particularviewpoint, as described above, may allow for arbitrary manipulation ofthe real-world objects in relation to one another and to an immersivevirtual reality world in general. Specifically, as will be described inmore detail below, individually-manipulable volumetric models of objectsmay be arbitrarily added to immersive virtual reality worlds, removedfrom immersive virtual reality worlds, replaced (i.e., swapped out withdifferent objects) within immersive virtual reality worlds, viewed fromarbitrary points of view within immersive virtual reality worlds, and soforth.

To this end, a virtual reality media provider system may include aconfiguration of synchronous video and depth capture devices (e.g.,video cameras, three-dimensional (“3D”) depth scanning hardware, etc.)disposed at fixed positions in a vicinity (e.g., within a line of sight)of a first object. In some examples, the first object may be located ina natural setting along with one or more additional objects.

As used herein, a “natural setting” broadly includes any setting that isnot used to specifically create a volumetric model of an object. Inother words, natural settings may include various types of settings(e.g., indoor settings, outdoor settings, artificially-created settings,nature-created settings, etc.) in which an object may be located (e.g.,along with other objects) for purposes other than for creating avolumetric model of the object. For example, a studio setting in whichan object is positioned in front of a “green screen” or other similarbackdrop in order to scan the object and create a volumetric model ofthe object may not be considered a natural setting, while anartificially-created set of a motion picture where people and objectsinteract while being filmed for a motion picture may be considered anatural setting.

In certain examples, natural settings may be associated with real-worldevents (i.e., events that take place in the real-world, as opposed totaking place only in a virtual world). For example, a real-world eventmay be a sporting event (e.g., a basketball game, an Olympic event,etc.), a concert (e.g., a rock concert in a large venue, a classicalchamber concert in an intimate venue, etc.), a theatrical presentation(e.g., a Broadway musical, an outdoor pageant, etc.), a large-scalecelebration (e.g., New Year's Eve on Times Square, Mardis Gras, etc.), arace (e.g., a stock-car race, a horse race, etc.), a political event(e.g., a presidential debate, a political convention, etc.), or anyother real-world event that may interest potential viewers. Thereal-world event may take place at any indoor or outdoor real-worldlocation.

The configuration of synchronous video and depth capture devicesdisposed at the fixed positions in the vicinity of the first object(e.g., within a natural setting) may be configured to capturetwo-dimensional (“2D”) video data and depth data for a surface of thefirst object while the first object is located in the natural settingalong with the one or more additional objects. As used herein, “2D videodata” may broadly include any data representative of how a real-worldsubject (e.g., a real-world scene, one or more objects within a naturalsetting, etc.) may appear over a particular time period and from atleast one vantage point of at least one device capturing the 2D videodata. 2D video data is not limited to any particular format, file type,frame rate, resolution, quality level, or other characteristic that maybe associated with various definitions and/or standards defining videoin the art. In certain examples, 2D video data may include a capturedsequence of images (e.g., high-resolution still images) representativeof an object within a natural setting over a particular time period. Asused herein, “depth data” may broadly include any data representative ofa position of a real-world subject (e.g., one or more objects within anatural setting) in 3D space. As will be described in more detail below,depth data may be captured based solely on 2D video data (e.g., bycombining 2D video data captured from different vantage points using asuitable depth capture technique) or by using techniques that mayrequire additional depth capture equipment and/or data such asspecialized depth capture devices that provide time-of-flight data,infrared imaging data, and the like. In certain examples, 2D video datamay be synchronized with depth data such that individually-manipulablevolumetric models of objects that incorporate the 2D video data and thedepth data across a period of time may be generated.

Accordingly, video and depth capture devices may capture 2D video dataand depth data in any suitable way and using any suitable devices as mayserve a particular implementation. Specifically, as will be described inmore detail below, in certain examples, video and depth capture devicemay consist of video cameras or other types of image capture devicesthat may capture 2D video data of objects within a natural setting frommultiple vantage points from which depth data for the surfaces of theobjects may be captured (e.g., derived) by using one or more depthcapture techniques (e.g., triangulation-based depth capture techniques)described herein. In other examples, as will also be described in moredetail below, video and depth capture devices may include video camerasor other types of image capture devices configured to capture the 2Dvideo data, as well as separate depth capture devices configured tocapture the depths of the surface of the objects using one or more ofthe depth capture techniques described below (e.g., time-of-flight-baseddepth capture techniques, infrared-based depth capture techniques,etc.). In the same or other examples, video and depth capture devicesmay include unitary devices that include video camera devices andspecialized depth capture devices combined together in single devicesthat are similarly configured to capture the depth data using one ormore depth capture techniques described here. Additionally, theconfiguration of synchronous video and depth capture devices maycontinuously capture the 2D video data and the depth data in time, suchthat the first object may be modeled in all four dimensions of space andtime.

As used herein, an “object” may broadly include anything that is visible(i.e., non-transparent) from a particular viewpoint, whether living orinanimate. For example, as will be described below, if the setting is areal-world event such as a basketball game, the first object for whosesurface the video and depth capture devices may capture 2D video dataand depth data may be a basketball being used for the game, while theadditional objects included in the natural setting with the basketballmay include objects such as a basketball court, a basketball standard(e.g., backboard, rim, net, etc.), a player or referee participating inthe game, and/or other objects associated with the basketball game.

In some examples, the video and depth capture devices may capture the 2Dvideo data and depth data in real-time (e.g., as the basketball game isbeing played) so that virtual reality media content representative ofthe real-world event (e.g., the basketball game) may be distributed tousers to experience live, as will be described below.

Based on the captured depth data and the captured 2D video data from thevideo and depth capture devices, the virtual reality media providersystem may distinguish the first object from a second object included inthe one or more additional objects located in the natural setting alongwith the first object. For instance, in the basketball game exampledescribed above, the virtual reality media provider system maydistinguish a basketball from a player holding the basketball.Techniques for distinguishing objects from other objects will bedescribed below.

Also based on the captured depth data and the captured 2D video datafrom the video and depth capture devices, the virtual reality mediaprovider system may generate an individually-manipulable volumetricmodel of the first object. An individually-manipulable volumetric modelof an object may include and/or be generated based both on 1) depth datarepresenting where and how the object is positioned in 3D space at aparticular time, or with respect to time over a particular time period,and on 2) synchronous 2D video data mapped onto a positional model(e.g., a wireframe model of the object derived from the depth data) torepresent how the object appeared at the particular time or with respectto time over the particular time period. As such,individually-manipulable volumetric models may be 3D models includingthree spatial dimensions or four-dimensional (“4D”) models that includethe three spatial dimensions as well as a temporal dimension.Additionally, the individually-manipulable volumetric model of the firstobject may be configured to be individually manipulated with respect toan immersive virtual reality world while a user of a media player deviceis experiencing the immersive virtual reality world using the mediaplayer device. For example, the immersive virtual reality world may bebased on virtual reality media content provided to the media playerdevice and representative of the immersive virtual reality world.

More specifically, as will be described below, one or moreindividually-manipulable volumetric models of objects within the naturalsetting may be combined into a volumetric data stream (e.g., a real-timevolumetric data stream) from which virtual reality media content may begenerated. In some examples, the generation of the volumetric datastream may be performed in real time such that users not physicallypresent in the natural setting (e.g., not attending a real-world eventsuch as a basketball game, not physically on location at a scene wherenews coverage is taking place, etc.) may be able to experience what ishappening in the natural setting and the actions of the objects withinthe natural setting live, in real time, via virtual reality mediacontent corresponding to the natural setting. Examples ofindividually-manipulable volumetric models of objects within naturalsettings, as well as volumetric data streams and techniques for creatingand distributing individually-manipulable volumetric models andvolumetric data streams will be described below.

Virtual reality media content representative of an immersive virtualreality world may be generated and/or provided to a media player deviceassociated with a user. For example, as will be described below, thevirtual reality media content may be generated from data within avolumetric data stream that includes individually-manipulable volumetricmodels of objects. The virtual reality media content may be generatedand/or provided by the virtual reality media provider system and/or byanother system operated by the virtual reality media provider or by aseparate entity (e.g., a virtual reality media content distributorassociated with the virtual reality media provider). While the user isexperiencing the immersive virtual reality world provided within thevirtual reality media content using the media player device, theindividually-manipulable volumetric model of the first object and/orindividually-manipulable volumetric models of other objects representedin the immersive virtual reality world may be individually manipulatedwith respect to one another and/or with respect to the immersive virtualreality world in general. For example, the individually-manipulablevolumetric models may be individually manipulated by a system generatingand/or providing the virtual reality media content (e.g., the virtualreality media provider system, a virtual reality media contentdistributor system, etc.), by the media player device presenting thevirtual reality media content, or by any other system as may serve aparticular implementation.

As used herein, individually-manipulable volumetric models of objectsmay each be “individually manipulated” with respect to each other and/orwith respect to an immersive virtual reality world by being processed(e.g., added, removed, modified, moved, replaced, rotated, graphicallyaltered, etc.) as a discrete unit independent of otherindividually-manipulable volumetric models of other objects in theimmersive virtual reality world and/or independently of the immersivevirtual reality world in general. For example, as described below, thevirtual reality media provider system may individually manipulate anindividually-manipulable volumetric model of an object within animmersive virtual reality world by inserting theindividually-manipulable volumetric model into the immersive virtualreality world (e.g., at any location within the immersive virtualreality world), removing the individually-manipulable volumetric modelfrom the immersive virtual reality world, replacing (e.g., swapping out)the individually-manipulable volumetric model with a differentindividually-manipulable volumetric model in the immersive virtualreality world, replacing (e.g., swapping out) a differentindividually-manipulable volumetric model of a different object with theindividually-manipulable volumetric model in the immersive virtualreality world, modifying (e.g., rotating, resizing, recoloring, shading,moving, etc.) the individually-manipulable volumetric model in theimmersive virtual reality world to make the individually-manipulablevolumetric model appear different or to be viewed from a differentviewpoint, or by otherwise manipulating the individually-manipulablevolumetric model in the immersive virtual reality world as may serve aparticular implementation.

In some examples, such as when the natural setting is associated with areal-world event, it may be desirable for the users who are notattending the real-world event to experience the real-world event live(e.g., in real time as the real-world event is occurring with as small adelay as possible). Accordingly, the virtual reality media providersystem may provide virtual reality media content representative of animmersive virtual reality world corresponding to the real-world event tomedia player devices in real time. Additionally or alternatively, thevirtual reality media provider system may provide the virtual realitymedia content representative of an immersive virtual reality worldcorresponding to a real-world event to media player devices in atime-shifted manner.

While data processing and data distribution may take a finite amount oftime such that it is impossible for a user to experience real-worldevents precisely as the real-world events occur, as used herein, anoperation (e.g., providing the virtual reality media content) isconsidered to be performed “in real time” when the operation isperformed immediately and without undue delay. Accordingly, a user maybe said to experience a real-world event in real time even if the userexperiences particular occurrences within the event (e.g., a particularshot in a basketball game) a few seconds or minutes after theoccurrences actually take place at the real-world event. Certain methodsand systems disclosed herein may be specially adapted to supportreal-time volumetric modeling and experiencing of immersive virtualreality worlds based on the natural setting. For example, powerfulhardware resources (e.g., multiple servers including multiple processingunits) may be employed to perform the immense processing required forreal-time creation and distribution of immersive virtual reality worldsbased on individually-manipulable volumetric models. Moreover,particular techniques for capturing 2D video data and depth data (e.g.,such as techniques described below) or for distinguishing and separatelymodeling different types of objects (e.g., static, dynamic, andbackground objects as described below) may further facilitate and/orenable the immense processing to be performed in real-time.

By creating and manipulating an individually-manipulable volumetricmodel of an object located in a natural setting, immersive virtualreality worlds based on real-world objects may be generated that are not“flat” (e.g., as in the flat conglomerate scenes described above), butthat are instead dynamic and manipulable on an object-by-object basis.As a result, specific objects represented in the immersive virtualreality world may be freely and independently manipulated, even in realtime, as may be directed by a user experiencing the immersive virtualreality world, or as directed by virtual reality content creators (e.g.,directors and/or producers of virtual reality media content programs)associated with the virtual reality media provider system. For example,a scene in a virtual reality television show that includes a vendingmachine associated with a first brand of soft drink may be replaced inthe scene by a vending machine associated with a second brand of softdrink (e.g., according to which brand may be commercially sponsoring thevirtual reality television show).

Similarly, in certain examples, this concept may even be applied toobjects that are moving and/or performing actions. For example, a firstactor in a virtual reality motion picture may be replaced by a secondactor in the virtual reality motion picture. Because the behavior of thefirst actor (e.g., one or more actions performed by the first actor) maybe captured and associated with (e.g., stored along with) theindividually-manipulable volumetric model of the first actor, the secondactor may be made to behave in the same way as the first actor (e.g., toperform the same one or more actions) in the virtual reality motionpicture without any need for the second actor to actually perform theone or more actions. For example, the individually-manipulablevolumetric model of the second actor may be manipulated to behave in thesame fashion as the individually-manipulable volumetric model of thefirst actor based on the captured behavior of the first actor even ifthe second actor has never performed the one or more actions of thefirst actor. Moreover, objects may be added to a scene (e.g., adding apassenger to a seat of a car that would otherwise be empty), removedfrom a scene, or otherwise manipulated as may serve a particularimplementation.

Additionally, by generating individually-manipulable volumetric modelsfor all the objects in a particular natural setting, theindividually-manipulable volumetric models may be presented in animmersive virtual reality world that may be viewed from a dynamicallyselectable viewpoint corresponding to an arbitrary location in thevicinity of the natural setting. For example, the dynamically selectableviewpoint may be selected by the user of the media player device whilethe user is experiencing the real-world event using the media playerdevice.

As used herein, an “arbitrary location” may refer to any point in spacein a vicinity of one or more objects for which individually-manipulablevolumetric models have been generated (e.g., in or around the naturalsetting). For example, arbitrary locations are not limited to fixedpositions where video and depth capture devices may be disposed, butalso include all the positions between the video and depth capturedevices and even places where video and depth capture devices may not beable to be positioned. Moreover, arbitrary locations may not be limitedto aligning with a viewing angle (i.e., an angle of capture) of anyvideo and depth capture device in the configuration of synchronous videoand depth capture device in the vicinity of the objects.

In some examples, such arbitrary locations (i.e., that do not directlyalign with a viewing angle of any video and depth capture device) maycorrespond to desirable viewpoints where cameras may not be able to bepositioned. For instance, in the basketball game example presentedabove, video and depth capture devices may not be allowed to bepositioned in the middle of the basketball court because the video anddepth capture devices would interfere with gameplay of the basketballgame. However, if individually-manipulable volumetric models of theobjects on and around the basketball court have been generated, a usermay dynamically select viewpoints from which to experience the game thatare in any arbitrary location on the basketball court. For example, theuser may dynamically select his or her viewpoint to follow thebasketball up and down the basketball court and experience thebasketball game as if standing on the basketball court in the middle ofthe action of the game. In other words, for example, while video anddepth capture devices may be positioned at fixed positions surroundingthe basketball court, but may not be positioned directly on the court soas not to interfere with gameplay of the basketball game, the user maydynamically select viewpoints from which to experience the game that arein any arbitrary location on the basketball court.

By creating and manipulating an individually-manipulable volumetricmodel of an object located in a natural setting that allows objects tobe individually manipulated as described herein, a virtual reality mediaprovider system may provide users with greater flexibility for tailoringvirtual reality media content programs to their preferences (e.g.,allowing users to select favorite actors and/or objects (e.g., cars,etc.) to be presented within particular scenes), and may provide moretailored and natural product placement advertising for sponsors.Moreover, the virtual reality media provider system may facilitate usersbecoming immersed in natural settings (e.g., real-world events, etc.) toan extent that may not be possible for people experiencing the naturalsettings using traditional media (e.g., television), traditional virtualreality media, or even by being physically present in the naturalsetting. For example, based on a particular manipulation of severalindividually-manipulable volumetric models of objects, a user may beable to experience a live basketball game as if running up and down thecourt with the players, or experience a live concert as if standing onstage next to the performers.

Various embodiments will now be described in more detail with referenceto the figures. The disclosed methods and systems may provide one ormore of the benefits mentioned above and/or various additional and/oralternative benefits that will be made apparent herein.

FIG. 1 illustrates an exemplary virtual reality media provider system100 (“system 100”) that may create and manipulate anindividually-manipulable volumetric model of an object located in anatural setting in accordance with principles described herein. Asshown, system 100 may include, without limitation, a data capturefacility 102, a data processing facility 104, a data distributionfacility 106, and a storage facility 108 selectively and communicativelycoupled to one another. It will be recognized that although facilities102 through 108 are shown to be separate facilities in FIG. 1,facilities 102 through 108 may be combined into fewer facilities, suchas into a single facility, or divided into more facilities as may servea particular implementation. Each of facilities 102 through 108 mayinclude or be housed in a device (e.g., having a single chassis) andlocated at a single location or distributed between multiple devicesand/or multiple locations as may serve a particular implementation. Eachof facilities 102 through 108 will now be described in more detail.

Data capture facility 102 may include any hardware and/or software(e.g., computing systems, video and depth capture equipment, softwareprograms, etc.) used for capturing data associated with attributes ofobjects in a natural setting. For example, data capture facility 102 mayinclude a configuration of synchronous video and depth capture devicessuch as 2D video cameras, 3D depth scanners, unitary devices (e.g.,combination video-depth capture devices configured to capture both 2Dvideo and associated depth data), and so forth. Examples of video anddepth capture devices will be described in more detail below. Datacapture facility 102 may be used to capture two-dimensional video dataand depth data for surfaces of objects in a natural setting in any waydescribed herein and/or as may serve a particular implementation.

Data processing facility 104 may include any hardware and/or software(e.g., computing systems, software programs, etc.) used for processingthe data captured by data capture facility 102, for distinguishing oneobject from another in the captured 2D video data and captured depthdata, and/or for generating individually-manipulable volumetric modelsof one or more objects in the natural setting. For example, dataprocessing facility 104 may include one or more server systems or othercomputing devices running specialized and/or general-purpose imageprocessing software, 3D modeling software, and so forth. Examples of howdata processing facility 104 may process captured data to distinguish afirst object from a second object included in one or more additionalobjects located in the natural setting along with the first object andto generate an individually-manipulable volumetric model of the firstobject will be described below. Data processing facility 104 may alsogenerate virtual reality media content representative of an immersivevirtual reality world corresponding to the natural setting based on theindividually-manipulable volumetric model.

Data distribution facility 106 may include any hardware and/or software(e.g., computing systems, networking systems, software programs, etc.)used for distributing data processed (e.g., generated) by dataprocessing facility 104 and/or for providing virtual reality mediacontent representative of the real-world event (e.g., virtual realitymedia content generated by data processing facility 104) as experiencedfrom a dynamically selectable viewpoint corresponding to an arbitrarylocation at the real-world event. To this end, data distributionfacility 106 may also receive data representative of user input (e.g.,selections of dynamically selectable viewpoints corresponding toarbitrary locations at the real-world event) from users experiencing thereal-world event using media player devices to present the virtualreality media content.

Storage facility 108 may maintain volumetric model data 110 and/or anyother data received, generated, managed, maintained, used, and/ortransmitted by facilities 102 through 106. Volumetric model data 110 mayinclude data representing individually-manipulable volumetric models(e.g., 3D models, 4D models, etc.) of various objects within the naturalsetting generated by data processing facility 104 from 2D video dataand/or depth data captured by data capture facility 102. As such, system100 may provide virtual reality media content representative of animmersive virtual reality world corresponding to the natural setting inwhich the objects within the natural setting may be manipulated invarious ways described herein (e.g., replacing, modifying, adding, orremoving objects within the immersive virtual reality world, etc.).Additionally, system 100 may provide virtual reality media contentrepresentative of an immersive virtual reality world corresponding tothe natural setting as viewed from a dynamically selectable viewpointcorresponding to an arbitrary location in the vicinity of the naturalsetting by manipulating and providing various individually-manipulablevolumetric models within volumetric model data 110 to different mediaplayer devices based on dynamically selectable viewpoints that areselected by different respective users of the media player devices.Storage facility 108 may further include any other data as may be usedby facilities 102 through 106 to create and manipulateindividually-manipulable volumetric models of objects located in anatural setting as may serve a particular implementation.

In some implementations, system 100 may perform some or all of theoperations for creating and manipulating an individually-manipulablevolumetric model of an object located in a natural setting in real time.For example, system 100 may capture the two-dimensional video data anddepth data for the surface of a first object, distinguish the firstobject from a second object, generate an individually-manipulablevolumetric model of the first object, and individually manipulate theindividually-manipulable volumetric model of the first object withrespect to an immersive virtual reality world represented in virtualreality media content in real time while the first object is located inthe natural setting.

FIG. 2 illustrates an exemplary implementation 200 of system 100 shownin FIG. 1. As shown, implementation 200 includes a configuration 202 ofsynchronous video and depth capture devices 204 (e.g., video and depthcapture devices 204-1 through 204-n). Implementation 200 furtherincludes a virtual reality media processing server 206 and a virtualreality media distribution server 208 communicatively coupled toconfiguration 202.

In configuration 202, synchronous video and depth capture devices 204(“capture devices 204”) may be disposed (i.e. located, installed, etc.)at fixed positions in a vicinity of a first object in any way that mayserve a particular implementation. For example, as will be illustratedand described below, configuration 202 may include capture devices 204at fixed positions surrounding the first object in the natural setting.For instance, in the basketball game example described above, capturedevices 204 may surround objects on a field of play (e.g., a basketballcourt) at a sporting event (e.g., the basketball game). Similarly,capture devices 204 may surround a stage of a theatrical performancebeing performed, a set of a movie or television program being filmed, orany other natural setting or one or more portions thereof as may serve aparticular implementation.

Each capture device 204 may include one or more devices or componentsconfigured to continuously capture 2D video and/or depth data as mayserve a particular implementation. For example, each capture device 204may include a first component (e.g., a video camera device) configuredto capture 2D video of objects at which the first component is directed(e.g., pointed), and a second component (e.g., a depth camera device, a3D imaging or 3D scanning device, etc.) configured to capture depth dataof objects at which the second component is directed. Is this example,the first component and the second component may be separate or discretedevices, but may be communicatively coupled and configured to work inconjunction with one another to simultaneously and synchronously captureboth the 2D video data and the depth data.

In other examples, each capture device 204 may comprise a unitaryvideo-depth capture device (e.g., a specially-designed video camera)that is configured to capture both the 2D video data and the depth data.In other words, both the 2D video data and the depth data may becaptured using the same unitary video-depth capture device. The unitaryvideo-depth capture device may be a commercially available orspecially-designed video camera capable of not only capturing video databut also detecting corresponding depth of objects represented in thevideo data using one of the depth capture techniques described herein oranother suitable technique. Similarly, as mentioned above, in exampleswhere a depth capture technique being used relies only on 2D video data(e.g., certain triangulation-based depth capture techniques), capturedevices 204 may not include any specialize depth capture equipment orcapability (e.g., time-of-flight equipment, infrared sensing equipment,etc.) but, rather, may only include video capture devices and/or othersimilar types of image capture devices.

In some examples, capture devices 204 may have a limited viewing angle(e.g., 90 degrees, 120 degrees, etc.) designed to capture data fromobjects in a specific area of the natural setting. For example, a ringconfiguration of capture devices 204 with limited viewing angles maysurround objects within a natural setting or one or more portionsthereof (e.g., objects on a basketball court at a basketball game, carsat different turns on a racetrack, etc.) and may be pointed inwardly tocapture data associated with the objects (e.g., positioned around thebasketball court or the different turns of the racetrack, and pointinginwardly to the basketball court or to the different turns of theracetrack, etc.). In the same or other examples, at least one particularcapture device 204 may have a 360-degree viewing angle to capture datafrom objects surrounding the particular capture device 204. For example,at least one of capture devices 204 may be a 360-degree cameraconfigured to capture and/or generate a 360-degree video image of thenatural setting around a center point corresponding to the 360-degreecamera.

As used herein, a 360-degree video image is any video image that depictsthe surroundings of a center point (e.g., a center point associated withthe location of one of capture devices 204 such as a 360-degree camera)on all sides along at least one dimension. For example, one type of360-degree video image may include a panoramic video image that depictsa complete 360-degree by 45-degree ring around a center pointcorresponding to the camera. Another type of 360-degree video image mayinclude a spherical video image that depicts not only the ring aroundthe center point, but an entire 360-degree by 180-degree spheresurrounding the center point on all sides. In certain examples, a360-degree video image may be based on a non-circular geometricstructure. For example, certain 360-degree video images may be based oncubes, rectangular prisms, pyramids, and/or other geometric structuresthat may serve a particular implementation, rather than being based onspheres.

The 360-degree camera may be configured to capture a very wide-anglevideo image (e.g., using one or more “fish-eye” lenses to capture aspherical or semi-spherical image) or to capture a plurality of rawvideo images from each of a plurality of segment capture cameras builtinto or otherwise associated with the 360-degree camera. In someexamples, the 360-degree camera may generate the 360-degree video imageof the natural setting by combining (e.g., stitching together) theplurality of video images captured by the segment capture cameras. Inother examples, the 360-degree camera may send raw video image data toone or more servers (e.g., virtual reality media processing server 206)and the raw video images may be combined into a 360-degree (e.g.,spherical) video image by the one or more servers.

Capture devices 204 within configuration 202 may be communicativelycoupled to one another (e.g., networked together) and/or communicativelycoupled to another device (e.g., virtual reality media processing server206). This may allow the devices to maintain synchronicity in time,position, angle, etc. so that individually-manipulable volumetric modelsof the objects in the natural setting may be properly generated. Forexample, capture devices 204 may send and receive timing signals toensure that each of capture device 204 captures corresponding data atthe same time and that the data captured by different capture devices204 may be timestamped with a universal time shared by all of capturedevices 204 in configuration 202.

Virtual reality media processing server 206 may perform any of the dataprocessing operations described herein. For example, virtual realitymedia processing server 206 may be associated with (e.g., may implementall or a portion of or may be contained within) data processing facility104 and/or storage facility 108 of system 100. As such, virtual realitymedia processing server 206 may receive captured data from configuration202 of capture devices 204 and may use the captured data to distinguisha first object from a second object located in the natural setting andgenerate an individually-manipulable volumetric model of the firstobject and/or the second object in any way that may serve a particularimplementation.

Virtual reality media distribution server 208 may perform any of thedata distribution operations described herein. For example, virtualreality media distribution server 208 may be associated with (e.g.,implementing all or a portion of, or being contained within) datadistribution facility 106 and/or storage facility 108 of system 100. Assuch, virtual reality media distribution server 208 may receive captureddata from configuration 202 and/or processed data (e.g., theindividually-manipulable volumetric model of the objects and/or virtualreality media content that includes the individually-manipulablevolumetric models) from virtual reality media processing server 206, andmay distribute the captured and/or processed data to other devices. Forexample, virtual reality media distribution server 208 may providevirtual reality media content representative of the natural setting(e.g., based on and/or including one or more individually-manipulablevolumetric models of objects within the natural setting) to media playerdevices associated with users (not explicitly shown in FIG. 2).

FIG. 3 illustrates an exemplary configuration 300 in which system 100operates to create and manipulate an individually-manipulable volumetricmodel of an object located in a natural setting. As illustrated in theexample of FIG. 3, a natural setting may be a setting of a real-worldevent such as a basketball game. More specifically, as shown inconfiguration 300, the natural setting may include a stage space 302 ofthe real-world event (e.g., a basketball court), which may be surroundedby inward-facing synchronous video and depth capture devices 304-i andmay surround at least one outward-facing video and depth capture device304-o (collectively referred to as “capture devices 304”). Capturedevices 304 may be configured to capture 2D video data and depth datafor surfaces of objects 306 within the natural setting (e.g., players,the basketball, etc.). In some examples, capture devices 304 capture 2Dvideo data and depth data for the surfaces of objects 306 in real time.Basketball 308 is specifically called out in configuration 300 as aparticular example of an object 306 because a detailed example ofcreating an individually-manipulable volumetric model will be providedbelow with respect to basketball 308.

As further shown in configuration 300, capture devices 304 may becommunicatively coupled by cables 310 and/or by other means (e.g.,wireless networking means) to one another and/or to one or more servers312 (e.g., real-time servers processing captured data representative ofthe real-world event of the basketball game in real time). Servers 312,in turn, are communicatively coupled by a network 314 to one or moremedia player devices associated with one or more respective users,including a media player device 316 associated with a user 318. Asshown, servers 312 may also be communicatively coupled to a time-shiftstorage 320. Certain components in configuration 300 will now bedescribed in more detail.

Stage space 302 may include any portion of a natural setting (e.g., thereal-world event of the basketball game) that is targeted by a virtualreality media provider as being of interest to potential virtual realityviewers (e.g., such as user 318). For example, if, as in the example ofFIG. 3, the natural setting includes a real-world event such as abasketball game, the natural setting may include the entire basketballarena where the game is taking place (e.g., including the seating areas,etc.) while stage space 302 may include only the basketball court itselfand the space above the basketball court where the game is played. Inother examples, stage space 302 may include a stage where performers(e.g., actors in a play, musicians at a concert, a set where aproduction is being filmed, etc.) are performing, or other relevantareas of interest (e.g., specific turns and/or the finish line on aracetrack) depending on the nature of the natural setting, the level ofuser interest in the natural setting, the financial resources andpriorities of the virtual reality media provider capturing the naturalsetting, and any other factors that may serve a particularimplementation.

In some examples, the fixed positions at the natural setting wherecapture devices 304 are disposed include fixed positions outside ofstage space 302 (e.g., off of the basketball court) while objects 306that capture devices 304 may be directed at and for whichindividually-manipulable volumetric models may be created andmanipulated may be within stage space 302 (e.g., on the basketballcourt).

Capture devices 304 may be the same or similar to capture devices 204,described above in relation to FIG. 2. As shown, capture devices 304 maybe disposed at fixed positions in and around the natural setting (e.g.,the real-world event of the basketball game) such as surrounding stagespace 302 (in the case of capture devices 304-i) and/or in the middle ofstage space 302 (in the case of capture device 304-o). Thus, asdescribed above, capture devices 304-i may have limited viewing anglesbut may be directed inward to continuously capture details of what ishappening in stage space 302. Conversely, capture device 304-o may be a360-degree outward facing synchronous video and depth capture device(e.g., a 360-degree camera) configured to continuously capture360-degree 2D video data and depth data for surfaces of objects 306within stage space 302, as well as for objects 306 visible around thevicinity of the natural setting that are outside of stage space 302. Forexample, capture device 304-o may continuously capture datarepresentative of objects in the spectator seating areas at the venue inwhich the basketball game is taking place. Because capture device 304-omay not be able to be positioned directly within stage space 302 (i.e.,because it would interfere with the basketball game), capture device304-o may be suspended above stage space 302 or otherwise positioned asmay serve a particular implementation.

A configuration of capture devices 304 may include any suitable numberof cameras as may serve a particular implementation. For example, thenumber and position of capture devices 304 may be determined based on atarget quality level a virtual reality media provider strives to provideand/or based on a minimum number of cameras to reasonably capture datafrom objects 306 from enough angles to be able to adequately generatethe individually-manipulable volumetric models of the surfaces ofobjects 306. In other words, even when objects 306 are dynamicallymoving around within stage space 302 such that one object 306 maycompletely or partially block the view of another object 306 from theangle of a first capture device 304, the number and placement of capturedevices 304 may ensure that a second capture device 304 will have abetter angle with which to capture data for the blocked object 306 thandoes the first capture device 304.

Objects 306 may include any objects within the vicinity of the naturalsetting (e.g., located at or around the real-world event of thebasketball game) inside or outside stage space 302. For example, objects306 may include people on the court (e.g., basketball players, referees,and other people on the basketball court), basketball 308, and/or otherliving and/or inanimate objects such as basketball standards (i.e.,backboards, rims, nets, etc.), the floor of the basketball court, peopleand/or furniture on the sidelines of the basketball game, spectators andseating areas surrounding the basketball court, and the like. A specificexample of how 2D video data and depth data may be captured and used tocreate an individually-manipulable volumetric model of basketball 308will be described below.

Servers 312 may include any components described herein that may performoperations for creating and manipulating an individually-manipulablevolumetric model of an object 306 located in the vicinity of the naturalsetting of the basketball game. For example, servers 312 may include aplurality of powerful server systems (e.g., having multiple graphicsprocessing units) that implement system 100 and/or any of the systems orfacilities described in relation to system 100 in FIG. 1 or 2 orhereafter. In particular, servers 312 may receive captured data fromcapture devices 304 and create individually-manipulable volumetricmodels of objects 306. In certain examples, servers 312 may receive thecaptured data and create the individually-manipulable volumetric modelsin real time, such that users not present at the natural setting (e.g.,not attending the basketball game) may be able to still experience thenatural setting live (i.e., in real time). Servers 312 may also providevirtual reality media content representative of an immersive virtualreality world (e.g., to media player device 316 over network 314), andmay individually manipulate (e.g., in real time) theindividually-manipulable volumetric models of objects 306 with respectto the immersive virtual reality world within the virtual reality mediacontent while user 318 is experiencing the immersive virtual realityworld using media player device 316.

Network 314 may include any provider-specific network (e.g., a cable orsatellite carrier network or a mobile telephone network), the Internet,wide area network, or any other suitable network. Data may flow betweenservers 312, or between servers 312 and media player device 316 usingany communication technologies, devices, media, and protocols as mayserve a particular implementation. For example, servers 312 maycommunicate with one another or with media player device 316 using anysuitable communication technologies, devices, media, and/or protocolssupportive of data communications, including, but not limited to, socketconnections, Ethernet, data bus technologies, data transmission media,communication devices, Transmission Control Protocol (“TCP”), InternetProtocol (“IP”), File Transfer Protocol (“FTP”), Telnet, HypertextTransfer Protocol (“HTTP”), HTTPS, Session Initiation Protocol (“SIP”),Simple Object Access Protocol (“SOAP”), Extensible Mark-up Language(“XML”) and variations thereof, Real-Time Transport Protocol (“RTP”),User Datagram Protocol (“UDP”), Global System for Mobile Communications(“GSM”) technologies, Code Division Multiple Access (“CDMA”)technologies, Evolution Data Optimized Protocol (“EVDO”), 4G Long TermEvolution (“LTE”), Voice over IP (“VoIP”), Voice over LTE (“VoLTE”),WiMax, Time Division Multiple Access (“TDMA”) technologies, ShortMessage Service (“SMS”), Multimedia Message Service (“MMS”), radiofrequency (“RF”) signaling technologies, wireless communicationtechnologies (e.g., Bluetooth, Wi-Fi, etc.), in-band and out-of-bandsignaling technologies, and other suitable communications technologies.While only one network 314 is shown to interconnect servers 312 andmedia player device 316 in FIG. 3, it will be recognized that thesedevices and systems may intercommunicate by way of multipleinterconnected networks as may serve a particular implementation.

Media player device 316 may be used by user 318 to access and experiencevirtual reality media content received from system 100 (e.g., fromservers 312). To this end, media player device 316 may include or beimplemented by any device capable of presenting a field of view of animmersive virtual reality world (e.g., an immersive virtual realityworld corresponding to the natural setting) and detecting user inputfrom user 318 to dynamically update a scene of the immersive virtualreality world presented within the field of view as user 318 experiencesthe immersive virtual reality world.

For example, the field of view may provide a window through which user318 may easily and naturally look around the immersive virtual realityworld. The field of view may be presented by media player device 316(e.g., on a display screen of media player device 316) and may includevideo depicting objects surrounding the user within the immersivevirtual reality world. Additionally, the field of view may dynamicallychange in response to user input provided by user 318 as user 318experiences the immersive virtual reality world. For example, mediaplayer device 316 may detect user input (e.g., moving or turning thedisplay screen upon which the field of view is presented) from user 318.In response, the field of view may display different objects and/orobjects seen from a different viewpoint (e.g., a viewpoint correspondingto the position of the display screen) in place of the objects seen fromthe previous viewpoint.

In some examples, media player device 316 may be configured to allowuser 318 to select respective virtual reality media content programs(e.g., associated with different natural settings and real-world events,as well as other types of virtual reality media content programs) thatuser 318 may wish to experience. In certain examples, media playerdevice 316 may download virtual reality media content programs that user318 may experience offline (e.g., without an active connection toservers 312). In other examples, media player device 316 may request andreceive data streams representative of virtual reality media contentprograms that user 318 experiences while media player device 316 remainsin active communication servers 312 (e.g., system 100) by way of network314.

To facilitate user 318 in experiencing virtual reality media content,media player device 316 may include or be associated with at least onedisplay screen (e.g., a head-mounted display screen built into ahead-mounted virtual reality device or a display screen of a mobiledevice mounted to the head of the user with an apparatus such as acardboard apparatus) upon which scenes of an immersive virtual realityworld may be displayed. Media player device 316 may also includesoftware configured to receive, maintain, and/or process datarepresentative of the immersive virtual reality world to present thescenes of the immersive virtual reality world on the display screens ofthe media player devices. For example, media player device 316 mayinclude dedicated, standalone software applications (e.g., mobileapplications) configured to process and present data representative ofimmersive virtual reality worlds on the displays. In other examples, thesoftware used to present the particular scenes of the immersive virtualreality worlds may include non-dedicated software such as standard webbrowser applications.

Media player device 316 may take one of several different form factors.For example, media player device 316 may include or be implemented by ahead-mounted virtual reality device (e.g., a virtual reality gamingdevice) that includes a head-mounted display screen, by a personalcomputer device (e.g., a desktop computer, laptop computer, etc.), by amobile or wireless device (e.g., a smartphone, a tablet device, a mobilereader, etc.), or by any other device or configuration of devices thatmay serve a particular implementation to facilitate receiving and/orpresenting virtual reality media content. Different types of mediaplayer devices (e.g., head-mounted virtual reality devices, personalcomputer devices, mobile devices, etc.) may provide different types ofvirtual reality experiences having different levels of immersiveness foruser 318.

To illustrate, FIG. 4 shows different form factors of media playerdevice 316 configured to facilitate user 318 in experiencing animmersive virtual reality world based on individually-manipulablevolumetric models of objects located in a natural setting according tomethods and systems described herein.

As one example, a head-mounted virtual reality device 402 may be mountedon the head of user 318 and arranged so that each of the eyes of user318 sees a distinct display screen 404 (e.g., display screens 404-1 and404-2) within head-mounted virtual reality device 402. In some examples,a single display screen 404 may be presented and shared by both eyes ofuser 318. In other examples, distinct display screens 404 withinhead-mounted virtual reality device 402 may be configured to displayslightly different versions of a field of view of an immersive virtualreality world (e.g., representative of the natural setting). Forexample, display screens 404 may be configured to display stereoscopicversions of the field of view that may be captured by one or morestereoscopic cameras to give user 318 the sense that the immersivevirtual reality world presented in the field of view isthree-dimensional. Display screens 404 may also be configured to fillthe peripheral vision of user 318, providing even more of a sense ofrealism to user 318.

Moreover, head-mounted virtual reality device 402 may include motionsensors (e.g., accelerometers), directional sensors (e.g.,magnetometers), orientation sensors (e.g., gyroscopes), and/or othersuitable sensors to detect natural movements (e.g., head movements) ofuser 318 as user 318 experiences the immersive virtual reality world.Thus, user 318 may provide input indicative of a desire to move thefield of view in a certain direction and by a certain amount in theimmersive virtual reality world by simply turning his or her head inthat direction and by that amount. In some examples, user 318 may use aphysical console or controller to dynamically select a dynamicallyselectable viewpoint corresponding to an arbitrary location within thenatural setting (e.g., a viewpoint on stage space 302) from which toexperience (e.g., look around) the immersive virtual reality worldcorresponding to the natural setting.

In some examples, system 100 may manipulate individually-manipulablevolumetric models of objects within an immersive virtual reality worldbased on decisions made on the server side (e.g., by content creatorsassociated with a virtual reality media provider operating system 100).However, in other examples, user 318 may be empowered to cause system100 to manipulate some or all of the individually-manipulable volumetricmodels from which the immersive virtual reality world is generated. Tothis end, head-mounted virtual reality device 402 may include a userinterface to allow user 318 to manipulate (e.g., in real time)individually-manipulable volumetric models of various objects presentedwithin an immersive virtual reality world. For example, the userinterface may allow user 318 to replace one object in the immersivevirtual reality world (e.g., a car, a character played by a particularactor, etc.) with another object (e.g., a truck, the character played bya different actor, etc.). Similarly, the user interface may allow user318 to remove or add particular objects, modify objects, or otherwisemanipulate objects within the immersive virtual reality world in any waydescribed herein and/or as may serve a particular implementation.

As another example of a media player device 316, a personal computerdevice 406 having a display screen 408 (e.g., a monitor) may be used byuser 318 to experience the immersive virtual reality world correspondingto the natural setting. Because display screen 408 may not provide thedistinct stereoscopic view for each of the user's eyes and/or may notfill the user's peripheral vision, personal computer device 406 may notprovide the same degree of immersiveness that head-mounted virtualreality device 402 provides. However, personal computer device 406 maybe associated with other advantages such as its ubiquity among casualvirtual reality users that may not be inclined to purchase or use ahead-mounted virtual reality device. In some examples, personal computerdevice 406 may allow a user to experience virtual reality content withina standard web browser so that user 318 may conveniently experience theimmersive virtual reality world without using special devices ordownloading special software.

User 318 may provide user input to personal computer device 406 by wayof a keyboard 410, a mouse 412, and/or any other such input device asmay serve a particular implementation. For example, user 318 may usemouse 412 or navigation keys on keyboard 410 to move the field of view(i.e., to look around the immersive virtual reality world) and/or todynamically select a viewpoint within the natural setting from which toexperience the immersive virtual reality world (i.e., to “walk” or “fly”around within the immersive virtual reality world). In certain examples,a combination of keyboard 410 and mouse 412 may be used. Personalcomputer device 406 may additionally include a user interface to allowfor manipulation of objects (e.g., by individually manipulating theindividually-manipulable volumetric models included within the immersivevirtual reality world) similar to the user interface described above inrelation to head-mounted virtual reality device 402.

As yet another example of a media player device 316, a mobile device 414having a display screen 416 may be used by user 318 to experience theimmersive virtual reality world corresponding to the natural setting.Mobile device 414 may incorporate certain advantages of bothhead-mounted virtual reality devices and personal computer devices toprovide the most versatile type of media player device for experiencingthe immersive virtual reality world. Specifically, like personalcomputer devices, mobile devices are extremely ubiquitous, potentiallyproviding access to many more people than dedicated head-mounted virtualreality devices. However, because many mobile devices are equipped withmotion sensors, directional sensors, orientation sensors, etc., mobiledevices may also be configured to provide user 318 with an immersiveexperience comparable to that provided by head-mounted virtual realitydevices. For example, mobile device 414 may be configured to dividedisplay screen 416 into two versions (e.g., stereoscopic versions) of afield of view and to fill the peripheral vision of user 318 when mobiledevice 414 is mounted to the head of user 318 using a relativelyinexpensive and commercially-available mounting apparatus (e.g., acardboard apparatus). In other embodiments, mobile device 414 mayfacilitate experiencing the immersive virtual reality world by receivingmovement-based user input at arm's length (i.e., not mounted to the headof user 318 but acting as a hand-held dynamic window for experiencingthe immersive virtual reality world), by receiving swipe gestures on atouchscreen, or by other techniques that may serve a particularembodiment.

Mobile device 414 may additionally include an interface to allow formanipulation of objects (e.g., by individually manipulating theindividually-manipulable volumetric models included within the immersivevirtual reality world) similar to the user interface described above inrelation to head-mounted virtual reality device 402.

While examples of certain media player devices have been described, theexamples are illustrative and not limiting. A media player device mayinclude any suitable device and/or configuration of devices configuredto facilitate receipt and presentation of virtual reality media contentaccording to principles described herein. For example, a media playerdevice may include a tethered device configuration (e.g., a tetheredheadset device) or an untethered device configuration (e.g., a displayscreen untethered from a processing device). As another example, ahead-mounted virtual reality media player device or other media playerdevice may be used in conjunction with a virtual reality controller suchas a wearable controller (e.g., a ring controller) and/or a handheldcontroller.

As mentioned above, it may be desirable for user 318 to experience anatural setting (e.g., a real-world event such as a basketball game) inreal time (e.g., at the same time the real-world event is occurring orafter a trivial period of delay). However, in certain examples, user 318may wish to experience the natural setting in a time-shifted manner,rather than in real time. For example, if the natural setting isassociated with a real-world event that begins at a particular time anduser 318 tunes in to the virtual reality media content representative ofthe real-world event fifteen minutes late, user 318 may wish toexperience the real-world event starting from the beginning (i.e.,starting fifteen minutes before the time that user 318 tunes into thevirtual reality media content representative of the real-world event).Alternatively, user 318 may be busy when the real-world event occurs andmay wish to experience the real-world event later (e.g., the followingday).

To this end, system 100 may store and maintain, subsequent to providingvirtual reality media content representative of the immersive virtualreality world in real time, a recording of a volumetric data stream thatincludes all of the individually-manipulable volumetric model of theobjects at the real-world event. Then, when user 318 later wishes toexperience the real-world event, system 100 may provide virtual realitymedia content corresponding to the real-world event to media playerdevice 316 based on the recording of the volumetric data stream.

Returning to FIG. 3 to illustrate, configuration 300 includes time shiftstorage 320. Time shift storage 320 may be implemented within system 100(e.g., within storage facility 108). Additionally, while time shiftstorage 320 is illustrated as a stand-alone component in configuration300, it will be understood that time shift storage 320 may be includedwithin servers 312 and/or or within any other server or system as mayserve a particular implementation. When user 318 desires to experience atime-shifted, rather than a real time, version of the real-world event,system 100 (e.g., servers 312) may request and receive datarepresentative of the volumetric data stream recorded in time shiftstorage 320 and provide virtual reality media content representative ofthe real-world event to media player device 316 in the same or a similarmanner as if user 318 were experiencing the real-world event inreal-time.

An example will now be provided to illustrate how system 100 maycontinuously capture 2D video data and depth data for a surface of afirst object located in a natural setting, continuously distinguish thefirst object from a second object located in the natural setting withthe first object, and then continuously generate and update anindividually-manipulable volumetric model of the first object.

In particular, FIG. 5 illustrates an exemplary technique 500 forcreating an individually-manipulable volumetric model of an exemplaryobject in a natural setting. As shown in FIG. 5, a natural setting 502may include an object 504. A configuration of synchronous video capturedevices 506 and a configuration of synchronous depth capture devices 508may, respectively, capture 2D video data 510 and depth data 512 for thesurface of object 504 within natural setting 502. For example, videocapture devices 506 and depth capture devices 508 may be disposed atfixed positions in a vicinity of object 504 (e.g., surrounding object504) in natural setting 502 such that 2D video data and depth data forthe entire surface of object 504 (i.e., from every angle and vantagepoint) may be captured. 2D video data 510 and depth data 512 may thenundergo volumetric modeling 514 to generate an individually-manipulablevolumetric model 516 (e.g., a 3D model, a 4D model, etc.) of object 504.Individually-manipulable volumetric model 516 may be included withindividually-manipulable volumetric models of other objects withinnatural setting 502 (not explicitly shown) in a volumetric data stream(e.g., a real-time volumetric data stream) representative of the naturalsetting, as will be described in more detail below.

Natural setting 502 may be any natural setting mentioned herein or thatmay serve a particular embodiment. To continue with the examplepresented above in relation to FIG. 3, for example, natural setting 502may be a real-world event including a basketball game. Similarly, object504 may be any visible (i.e. nontransparent) object mentioned herein orthat may otherwise be present within natural setting 502. For example,object 504 may be animate (e.g., a person or an animal) or inanimate, asolid, a liquid, or a non-transparent gas (e.g., fog generated from afog machine at a concert), etc. In this example, as shown, object 504 isbasketball 308.

Video capture devices 506 and depth capture devices 508 may be the sameor similar to other video and depth capture devices described herein,such as capture devices 204 and/or 304. While only one video capturedevice and one depth capture device is illustrated in FIG. 5, it will beunderstood that each capture device may represent a configuration ofcapture devices that may surround object 504 to capture data for thesurface of object 504 from all sides (e.g., such as shown by capturedevices 304 in FIG. 3). As shown, video capture devices 506 and depthcapture devices 508 may be standalone capture devices (e.g., videocameras and 3D depth scanning devices, respectively). Alternatively, asdescribed above, video capture devices 506 and depth capture devices 508may be integrated into unitary video-depth capture devices configured tocapture both 2D video data and depth data using the same unitaryvideo-depth capture devices. In some examples, as mentioned above, depthdata may be determined based solely on 2D video data (e.g., 2D videodata from different vantage points) such that depth capture devices 508may represent the same video cameras and/or other types of image capturedevices represented by video capture devices 506.

2D video data 510 may be captured by video capture devices 506 and mayinclude image or texture data representative of visible characteristics(e.g., color, shading, surface texture, etc.) of the surface of object504 from all perspectives. For example, as illustrated in FIG. 5, 2Dvideo data 510 may represent visible characteristics of various sections(e.g., small areas) of the surface of basketball 308 as the sectionsappear from various vantage points of various video capture devices 506.For illustrative purposes, 2D video data 510 in FIG. 5 shows a pluralityof 2D images associated with various random sections of the surface ofbasketball 308 from a single vantage point. However, it will beunderstood that 2D video data 510 may include data associated with aplurality of vantage points surrounding basketball 308 and may becaptured, packaged, stored, formatted, and transmitted in any way thatmay serve a particular embodiment. For example, 2D video data 510 may bedelivered to volumetric modeling 514 with detailed information (e.g.,metadata) indicating temporal and spatial information, such as when the2D video data was captured, where the 2D video data was captured, etc.

Similarly, depth data 512 may be captured by depth capture devices 508and may comprise depth data representative of spatial characteristics(e.g., locational coordinates, etc.) of the surface of object 504 fromall perspectives. For example, as illustrated in FIG. 5, depth data 512may include captured data representative of depth characteristics ofvarious sections (e.g., small areas) of the surface of basketball 308such that a wireframe model of basketball 308 may be generated (e.g.,stitched together) based on the depth data captured from various vantagepoints associated with each depth capture device 508. Depth data 512 maybe captured, packaged, stored, formatted, and transmitted in any waythat may serve a particular embodiment. For example, depth data 512 maybe delivered to volumetric modeling 514 with detailed information (e.g.,metadata) indicating temporal and spatial information, such as when thedepth data was captured, where the depth data was captured, etc.

Depth data 512 may be determined by depth capture devices 508 using anytechnique or modality that may serve a particular implementation. Inparticular, certain depth capture techniques may be used to increase thetime efficiency of the depth capture (i.e., by minimizing capture and/orprocessing time) to facilitate generating individually-manipulablevolumetric models in real time.

For example, depth capture devices 508 may capture depth data 512 byusing a stereoscopic triangulation depth capture technique. In thistechnique, depth capture devices 508 may be configured to capture 2Dvideo data (i.e., depth capture devices 508 may be one and the same asvideo capture devices 506). The stereoscopic triangulation depth capturetechnique may include a first depth capture device 508 capturing 2Dvideo data of points on the surface of object 504 from a first angle anda second depth capture device 508 capturing 2D video data of the pointson the surface of object 504 from a second angle. The depth of thepoints on the surface of object 504 are triangulated based on the firstangle, the second angle, and on a predetermined distance (i.e., a knowndistance based on the configuration of depth capture devices 508)between the first depth capture device 508 and the second depth capturedevice 508.

In the same or other examples, depth capture devices 508 may capturedepth data 512 by using a time-of-flight depth capture technique. Forexample, depth capture devices 508 may use a radar-based rangingtechnique (e.g., laser radar) using electromagnetic pulses, asonar-based ranging technique using sound pulses, and/or any other typeof ranging technique as may serve a particular implementation. In thetime-of-flight technique, each depth capture device 508 may generate apulse (e.g., an electromagnetic pulse, a sound pulse, etc.) from asource associated with the depth capture device 508 at a particulartime, and may be specially configured to measure a total transit timefor the pulse to travel from the pulse source to points on the surfaceof object 504 (i.e., to travel to object 504), and, after beingreflected by the surface of object 504, to travel from the points on thesurface of object 504 to a pulse detector associated with the depthcapture device 508 (i.e., to return back to the depth capture device508). Based on the total transit time and the known speed of the pulse(e.g., the speed of light, the speed of sound, etc.), a depth of each ofthe points on the surface of object 504 may thus be determined.

In the same or other examples, depth capture devices 508 may capturedepth data 512 by using an infrared pattern analysis depth capturetechnique. In this technique, an infrared pattern emitter device (i.e.,associated with or separate from depth capture devices 508) may projecta random scatter (i.e., a pattern) of randomly-sized infrared dots ontosurfaces of various objects within natural setting 502, including object504. A first depth capture device 508 may be configured with infraredsensing capability such that the first depth capture device 508 maydetect the random scatter of randomly-sized infrared dots projected ontothe surfaces of the objects from a first angle. Similarly, a seconddepth capture device 508 similarly configured with infrared sensingcapability may detect the random scatter of randomly-sized infrared dotsprojected onto the surfaces of the objects from a second angle. Thedepth of the surfaces of the objects may then be triangulated based onthe first angle, the second angle, and on a predetermined distance(i.e., a known distance based on the configuration of depth capturedevices 508) between the first depth capture device 508 and the seconddepth capture device 508.

Because real-time depth detection in a non-controlled, real-worldenvironment may be difficult and inexact, in some examples, a pluralityof different depth capture techniques may be employed (e.g., such as thedepth capture techniques described above). Subsequently, depth dataobtained using each of the depth capture techniques employed may becombined to determine the most accurate depth data for objects withinnatural setting 502 possible.

Concurrently with or subsequent to video capture devices 506 and depthcapture devices 508 capturing 2D video data 510 and depth data 512 forobject 504, volumetric modeling 514 may distinguish object 504 from oneor more additional objects located in natural setting 502 along withobject 504. Volumetric modeling 514 may distinguish objects using one ormore exemplary techniques described below or using any technique thatmay serve a particular implementation. In particular, certain techniquesfor distinguishing objects may be used to increase the time efficiencyof the distinguishing (i.e., by minimizing processing time) tofacilitate the generation of individually-manipulable volumetric modelsin real time.

For example, FIG. 6 shows exemplary techniques 600 (e.g., techniques600-1 through 600-3) for distinguishing object 504 (i.e., basketball308) located in natural setting 502 from a second object 602 located innatural setting 502 along with object 504. On the left side of FIG. 6,object 504 is illustrated within natural setting 502 along with object602. While drawn in FIG. 6 as a simple cube, object 602 may representany other object that may be located within natural setting 502. Forexample, since natural setting 502 is a basketball game in this example,object 602 may represent a basketball standard, the floor of thebasketball court, a player holding the ball, or several backgroundobjects (e.g., spectators in an audience of the basketball game) seenbehind object 504 (i.e., basketball 308) at certain times and/or fromcertain vantage points during the basketball game. Regardless of whatobject 602 represents, volumetric modeling 514 may be configured todetermine that object 602 is a separate object (i.e., such that object602 should be associated with a separate individually-manipulablevolumetric model) from object 504, even though, from 2D video capturedfrom the vantage point illustrated in FIG. 6, objects 602 and 504 maypartially or completely overlap and/or otherwise appear to be part of asingle object.

In order to distinguish object 504 from object 602, a first techniquethat system 100 (e.g., volumetric modeling 514) may employ isidentifying objects 504 and 602 based on 2D video data 506, and thendetermining that object 504 as identified is different from object 602as identified. Specifically, system 100 may use object recognitionalgorithms to identify (e.g., recognize) object 504 as a basketball andobject 602 as, for example, a human being (e.g., a basketball player).System 100 may be programmed to recognize that basketballs and humanbeings are separate objects, even when a human being may be holding abasketball or positioned behind the basketball. Accordingly, based onthese identifications (i.e., object recognitions), system 100 maydistinguish object 504 from object 602.

Another technique that system 100 may use to distinguish object 504 fromobject 602 is illustrated by technique 600-1. In technique 600-1,shading is used to indicate relative depths captured from differentobjects. Accordingly, as shown, object 504 is lightened to indicate thatcaptured depth data 512 indicates that object 504 may be relativelyclose, while object 602 is darkened to indicate that captured depth data512 indicates that object 602 may be significantly farther away inthree-dimensional space than may appear to be the case judging only from2D video data 510. Thus, based on captured depth data 512, system 100may determine that object 504 is located at least a predetermineddistance away from object 602, which may indicate that objects 504 and602 are likely to be different objects.

Another technique that system 100 may use to distinguish object 504 fromobject 602 is illustrated by technique 600-2. In technique 600-2, thearrow indicates that object 504 may be moving (e.g., moving to theright) in relation to object 602. As such, system 100 may determine,based on 2D video data 510, that object 504 is moving in relation toobject 602, which system 100 may use as an indication that objects 504and 602 are likely to be separate, independent objects.

Yet another technique that system 100 may use to distinguish object 504from object 602 is illustrated by technique 600-3. In technique 600-3,objects 504 and 602 are viewed from a different vantage point than thevantage point shown on the left side of FIG. 6. Specifically, theoriginal vantage point showing the objects overlapping may come from afirst video capture device 506, while the vantage point shown intechnique 600-3 may represent the vantage point of a second videocapture device 506 in a different fixed position in the vicinity ofobjects 504 and 602. From the vantage point illustrated in technique600-3, distinguishing object 504 from object 602 may be trivial becausethe objects are clearly distinct and do not overlap. Thus, even if 2Dvideo data 510 indicates that objects 504 and 602 overlap from a firstvantage point of a first video capture device 506, system 100 maydetermine that 2D video data 510 also indicates that objects 504 and 602do not overlap from a second vantage point of at least one other videoand depth capture device (e.g., the second video capture device 506).System 100 may thus determine that objects 504 and 602 are likely to bedistinct objects.

It will be noted that system 100 may use any or all of the techniquesdescribed above individually or in combination with one another, as wellas with other techniques not explicitly described herein, to distinguishone object from another in a natural setting.

Returning to FIG. 5, concurrent with or subsequent to the capturing of2D video data 510 and depth data 512 and/or the distinguishing of object504 from other objects (e.g., object 602) within natural setting 502,volumetric modeling 514 may process 2D video data 510 together withdepth data 512 to generate individually-manipulable volumetric model 516of object 504 in any suitable way. For example, volumetric modeling 514may combine depth data 512 (e.g., using temporal and spatial metadataincluded with depth data 512) to create (e.g., stitch together) awireframe model of basketball 308, as shown in the drawing representingdepth data 512. Volumetric modeling 514 may then map 2D video data 510onto the wireframe model by matching temporal and spatial metadataincluded with 2D video data 510 with the temporal and spatialinformation included with depth data 512.

In this way, volumetric modeling 514 may generateindividually-manipulable volumetric model 516 (e.g., a 3D model, a 4Dmodel, etc.) of object 504 from 2D video data 510 and depth data 512.Individually-manipulable volumetric model 516 may be included withindividually-manipulable volumetric models of other objects withinnatural setting 502 in a volumetric data stream (e.g., a real-timevolumetric data stream) representative of natural setting 502, as willbe described in more detail below. However, eachindividually-manipulable volumetric model of each object within naturalsetting 502 may be individually and independently manipulable inrelation to the other individually-manipulable volumetric models of theother objects within natural setting 502. Accordingly, based on thevolumetric data stream including individually-manipulable volumetricmodel 516 and other individually-manipulable volumetric models for otherobjects within natural setting 502, virtual reality media content may begenerated and provided to arbitrarily and individually manipulate any ofthe objects within natural setting 502 (e.g., including objects 504 and602) in any way described herein. For example, virtual reality mediacontent may be provided allowing a user to view natural setting 502 inreal time from any arbitrary location within natural setting 502.

By generating distinct, individually-manipulable volumetric models ofeach object within natural setting 502 by combining depth data togenerate wireframe models of individual objects at specific points inspace in the vicinity of the objects within natural setting 502 andmapping 2D video data onto the wireframe models to createindividually-manipulable volumetric models of the objects within naturalsetting 502, objects within an immersive virtual reality world may beindividually and independently manipulated in various ways that may notbe possible (or may be extremely inefficient and/or processingintensive) by trying to manipulate 2D video data alone.

FIG. 7 illustrates an exemplary dataflow 700 for creating andmanipulating an individually-manipulable volumetric model of an object(e.g., object 504) located in a natural setting (e.g., natural setting502). The data in dataflow 700 may be generated, processed, distributed,etc., in any way described herein or as may serve a particularimplementation. As shown in FIG. 7, 2D video-depth data 702 (e.g., 2Dvideo-depth data 702-1 through 702-n and 702-o) may flow into volumetricmodeling 514, where static object depth modeling 704, static objectimage mapping 706, dynamic object depth modeling 708, dynamic objectimage mapping 710, and external modeling 712 may process 2D video-depthdata 702 to generate (e.g., in real time) a volumetric data stream 714(e.g., a real-time volumetric data stream). Virtual reality mediacontent 716, which may be generated based on volumetric data stream 714,may then be provided to a media player device.

Volumetric data stream 714 may include one or moreindividually-manipulable volumetric models 718 (e.g.,individually-manipulable volumetric models 718-1 through 718-x). Forexample, individually-manipulable volumetric models 718 may be 3D models(e.g., models representative of the three spatial dimensions of anobject) that may be manipulated to move and/or perform particularactions over a period of time (e.g., a 3D model of a first personreplacing a 4D model of a second person and performing the same actionsas will be described in more detail below). In other examples,individually-manipulable volumetric models 718 may be 4D models (e.g.,models representative of the three spatial dimensions of an object aswell as a temporal dimension) that may move and/or perform particularactions inherent to the 4D model itself. Virtual reality media content716 may include one or more individually-manipulable volumetric models718 (i.e., individually-manipulable volumetric models 718-5, 718-6, and718-x), which may be individually-manipulated within volumetric datastream 714 before distribution (e.g., by system 100) and/or afterdistribution (e.g., by the media player devices).

2D video-depth data 702 may represent captured 2D video data andcaptured 2D depth data from a plurality of video and depth capturedevices such as capture devices 204 (see FIG. 2), capture devices 304(see FIG. 3), or capture devices 506 and 508 (see FIG. 5). For example,2D video-depth data 502-1 may include 2D video data (e.g., similar to 2Dvideo data 510) and depth data (e.g., similar to depth data 512)captured by a first video and depth capture device. 2D video-depth data702-2 may include 2D video data and depth data captured by a secondvideo and depth capture device (e.g., a video and depth capture devicecapturing data representative of objects from a different vantage pointthan the first video and depth capture device), and so forth for 2Dvideo-depth data 702-3 through 702-n. 2D video-depth data 702-o mayinclude 2D video data and/or depth data captured by an outward facingvideo and depth capture device (e.g., a 360-degree outward facingsynchronous video and depth capture device such as capture device 304-oin FIG. 3).

As described above, volumetric modeling 514 may perform data processingon 2D video-depth data 702 to generate a volumetric data streamrepresentative of individually-manipulable volumetric models 718 of theobjects within the natural setting. More specifically, volumetricmodeling 514 may generate individually-manipulable volumetric models 718of at least three categories of objects: 1) static objects within thenatural setting (e.g., a floor of a basketball court, basketballstandards, etc.), 2) dynamic objects within the natural setting (e.g.,players and referees moving around on the basketball court, a basketballbeing used by the players in a basketball game, etc.), and 3) externalobjects within (or in the vicinity of) the natural setting (e.g.,objects outside the basketball court stage space such as spectatorswatching the basketball game, etc.). System 100 may obtain significantefficiency gains by differentiating these categories of objects andgenerating individually-manipulable volumetric models for the objectsseparately, rather than treating the different categories of objectsequally. For example, by differentiating static, dynamic, and externalobjects as described below, system 100 may obtain efficiency gains thatfacilitate and/or enable system 100 to perform the immense processingrequired to generate and provide individually-manipulable volumetricmodels of objects within the natural setting in real time.

Static object depth modeling 704 may model (e.g., create wireframe depthmodels for) one or more static objects within the natural setting basedon depth data within 2D video-depth data 702. For example, static objectdepth modeling 704 may determine, based on depth data, that a basketballstandard is statically located at a particular location in the spaceabove the basketball court, that the basketball standard is distinctfrom players and/or a basketball that may occasionally touch thebasketball standard, and that the basketball standard is distinct fromother objects seen behind the basketball standard (e.g., in thebackground) when the basketball standard is viewed from differentvantage points. With these determinations, static object depth modeling704 may generate a depth model (e.g., a wireframe model) of thebasketball standard that may not yet include any color or video data,but that may represent a location in a 3D space representative ofnatural setting where the basketball standard is positioned.

Static object image mapping 706 may map textures, colors, etc. onto thedepth model of static objects (e.g., the basketball standard) generatedby static object depth modeling 704. For example, static object imagemapping 706 may map the textures, colors, and so forth based on 2D videodata within 2D video-depth data 702. As such, completeindividually-manipulable volumetric models 718 of the static objects maybe included within volumetric data stream 714. Because the staticobjects may not change often or at all, individually-manipulablevolumetric models 718 of the static objects may be processed and updatedirregularly or on an as-needed basis in order to conserve processingresources in system 100.

Dynamic object depth modeling 708 and dynamic object image mapping 710may perform similar respective functions as static object depth modeling704 and static object image mapping 706 for dynamic objects (e.g.,objects determined to be dynamically moving in real time). However,because the dynamic object may be continuously in flux (e.g., movingaround within the natural setting), individually-manipulable volumetricmodels 718 of the dynamic objects may be updated much more regularly inorder to keep the volumetric data stream up-to-date with what isoccurring within the natural setting.

External modeling 712 also may perform similar functions as the depthmodeling and image mapping operations described above for externalobjects (e.g., background objects that are not within the stage space)such as those represented in 2D video-depth data 702-o. Because theexternal objects may add ambience and realism to a virtual realityexperience but may not be a primary focus of the experience for manyusers, external modeling 712 may update models for external objectsirregularly. Additionally, because 2D video-depth data 702-o may includecaptured data from only one or a limited number of vantage points (i.e.,vantage points that do not surround the external objects to capture datafrom every vantage point), external modeling 712 may generate a 2D model(e.g., a model that incorporates little or no depth data but is justbased on 2D video data) of the external objects, or a volumetric modelthat includes less detail than the individually-manipulable volumetricmodels 718 of, for example, the dynamic objects within a stage space ofthe natural setting.

As described above, volumetric data stream 714 may includeindividually-manipulable volumetric models 718 (e.g., 3D models, 4Dmodels) and/or other models (e.g., 2D models) of each of the staticobjects, dynamic objects, and external objects within a particularnatural setting. Accordingly, virtual reality media content 716 may beprovided based on volumetric data stream 714 to present objects thathave been individually, independently, and/or dynamically manipulated.

To illustrate, FIG. 8 shows an exemplary virtual reality experience 800in which user 318 is presented with exemplary virtual reality mediacontent representative of an immersive virtual reality world thatcorresponds to a natural setting containing an object for which anindividually-manipulable volumetric model has been generated.Specifically, virtual reality media content 802 is presented within afield of view 804 that shows a part of an immersive virtual realityworld 806 from a viewpoint corresponding to an arbitrary location withinimmersive virtual reality world 806 (e.g., on the basketball courtdirectly in front of a basketball player dribbling a basketball). Asshown, an individually-manipulable volumetric model of the player ispresented within virtual reality media content 802 along withindividually-manipulable volumetric model 516 (e.g., theindividually-manipulable volumetric model of the basketball describedabove in relation to FIG. 5). Immersive virtual reality world 806 maycorrespond to (e.g., may be based on) a natural setting of a basketballgame and may be available for the viewer to experience by providing userinput (e.g., head movements, keyboard input, etc.) to look around and/orto move around (i.e., dynamically select a viewpoint from which toexperience) immersive virtual reality world 806.

In FIG. 8, immersive virtual reality world 806 is illustrated as asemi-sphere, indicating that user 318 may look in any direction withinimmersive virtual reality world 806 that is substantially forward,backward, left, right, and/or up from the viewpoint of the locationunder the basketball standard that user 318 has currently selected. Inother examples, immersive virtual reality world 806 may include anentire 360-degree by 180-degree sphere such that user 318 may also lookdown. Additionally, user 318 may move around to other locations withinimmersive virtual reality world 806 (i.e., dynamically selectingdifferent dynamically selectable viewpoints within the natural setting).For example, user 318 may select a viewpoint at half court, a viewpointfrom the free-throw line facing the basketball standard, a viewpointsuspended above the basketball standard, or the like.

Additionally, objects presented within immersive virtual reality world806 may be individually manipulated in various ways as described hereinand/or as may serve a particular implementation. For example, FIG. 9illustrates exemplary manipulations that may be performed onindividually-manipulable volumetric model 516 with respect to immersivevirtual reality world 806 while user 318 is experiencing immersivevirtual reality world 806. In particular, in FIG. 9, field of view 804is illustrated as having various different versions of virtual realitymedia content 802 (e.g., virtual reality media content 802-1 through802-3) where individually-manipulable volumetric model 516 has beenmanipulated with respect to immersive virtual reality world 806 indifferent ways to illustrate certain potential ways thatindividually-manipulable volumetric model 516 may by individuallymanipulated.

For example, as shown in virtual reality media content 802-1, theindividually manipulating of individually-manipulable volumetric model516 with respect to immersive virtual reality world 806 may includereplacing individually-manipulable volumetric model 516 included withinimmersive virtual reality world 806 with an individually-manipulablevolumetric model of a different object within immersive virtual realityworld 806. in virtual reality media content 802-1,individually-manipulable volumetric model 516, which was in the hand ofthe basketball player in virtual reality media content 802 in FIG. 8,has been replaced by an individually-manipulable volumetric model 902 ofa different basketball (e.g., a lighter colored basketball).

In certain examples, as mentioned above, the object being replacedwithin an immersive virtual reality world may be a first person (e.g., aparticular actor playing a role in a virtual reality motion picture ortelevision show, a particular athlete in a sporting event, etc.) and theobject replacing the first person may be a second, different person(e.g., a different actor, a different athlete, etc.). Because system 100may have individually-manipulable volumetric models for both the firstperson and the second person, and because the individually-manipulablevolumetric model for the first person may be configured to perform aparticular action (i.e., acting in the role, playing the sport, etc.),system 100 may replace the individually-manipulable volumetric model ofthe first person with the individually-manipulable volumetric model ofthe second person such that the individually-manipulable volumetricmodel of the second person performs the same action in the immersivevirtual reality world that the individually-manipulable volumetric modelof the first person would perform if not replaced (i.e., acting in therole in the same way as the first person, playing the sport in the sameway as the first person, etc.).

More specifically, for example, system 100 may capture behavioral datafrom the first person with respect to various points on the surface ofthe first person (e.g., points corresponding to various parts of thefirst person's body), such as data representative of the person's armlifting up above the head, a hand opening or shutting, a leg taking astep, etc. Then, to replace the individually-manipulable volumetricmodel of the first person with the individually-manipulable volumetricmodel of the second person, the individually-manipulable volumetricmodel of the second person may be manipulated such that various pointson the surface of the second person (e.g., corresponding to the pointson the surface of the first person) appear to perform the same actionsas performed by the first person (e.g., lifting the arm above the head,opening or shutting the hand, taking the step, etc.). Accordingly, auser may select a favorite actor to play in any role or a favoriteathlete to play in any sporting event, etc., even though the selectedactor and/or athlete may never have actually performed the actionscorresponding to the role and/or the sporting event.

In other examples, as shown in virtual reality media content 802-2, theindividually manipulating of individually-manipulable volumetric model516 with respect to immersive virtual reality world 806 may includeinserting individually-manipulable volumetric model 516 into immersivevirtual reality world 806 at a particular location within immersivevirtual reality world 806. For example, an individually-manipulablevolumetric model may be inserted into a scene corresponding to a naturalsetting in which the object represented by the individually-manipulablevolumetric model is not (and/or never was) located, or may bearbitrarily duplicated in a scene corresponding to a natural setting inwhich the object is located. In virtual reality media content 802-2,individually-manipulable volumetric model 516 has not been modified inthe left hand of the player, but an additional individually-manipulablevolumetric model 516 has been added to the right hand of the player. Inthis way, individually-manipulable volumetric model 516 (or any otherindividually-manipulable volumetric model) may be arbitrarily added toimmersive virtual reality world 806 in any location within immersivevirtual reality world 806.

In yet other examples, as shown in virtual reality media content 802-3,the individually manipulating of individually-manipulable volumetricmodel 516 with respect to immersive virtual reality world 806 mayinclude removing individually-manipulable volumetric model 516 fromimmersive virtual reality world 806. For example, in virtual realitymedia content 802-3, individually-manipulable volumetric model 516 hasbeen individually and independently removed. In other words, as shown,individually-manipulable volumetric model 516 has been removed (i.e.,the basketball is no longer in the player's hand) whileindividually-manipulable volumetric models of other objects in immersivevirtual reality world 806 such as the individually-manipulablevolumetric model of the basketball player himself remain unaffected.More specifically, because the player and the basketball are eachrepresented by distinct individually-manipulable volumetric modelswithin immersive virtual reality world 806, removingindividually-manipulable volumetric model 516 representing thebasketball object has not altered the individually-manipulablevolumetric model of the player, as indicated by a hand 904 of the playerthat is now fully visible because the basketball is no longer present toobstruct hand 904. If the objects (the basketball and the player) werenot associated with individually-manipulable volumetric models, but,rather, the 2D video of the basketball were merely removed or otherwiseedited out of immersive virtual reality world 806, hand 904 may be fullyor partially removed from the scene along with the basketball.

The manipulations described herein and specifically illustrated inrelation to FIG. 9 are exemplary only. Once various objects within anatural setting have been modeled and associated withindividually-manipulable volumetric models, numerous other manipulationsmay be possible as may serve a particular implementation. Suchmanipulations are within the spirit and scope of this disclosure even ifnot explicitly mentioned herein.

FIG. 10 illustrates an exemplary method 1000 for creating andmanipulating an individually-manipulable volumetric model of an objectlocated in a natural setting. While FIG. 10 illustrates exemplaryoperations according to one embodiment, other embodiments may omit, addto, reorder, and/or modify any of the operations shown in FIG. 10. Oneor more of the operations shown in FIG. 10 may be performed by system100 and/or any implementation thereof.

In operation 1002, a virtual reality media provider system that includesa configuration of synchronous video and depth capture devices disposedat fixed positions in a vicinity of a first object may capture 2D videodata and depth data for a surface of the first object. In some examples,the virtual reality media provider system may capture the 2D video dataand depth data while the first object is located in a natural settingalong with one or more additional objects. Operation 1002 may beperformed in any of the ways described herein.

In operation 1004, the virtual reality media provider system maydistinguish the first object from a second object included in the one ormore additional objects located in the natural setting along with thefirst object. For example, the virtual reality media provider system maydistinguish the first object from the second object based on thecaptured depth data and captured 2D video data captured in operation1002. Operation 1004 may be performed in any of the ways describedherein.

In operation 1006, the virtual reality media provider system maygenerate an individually-manipulable volumetric model of the firstobject. For example, the virtual reality media provider system maygenerate the individually-manipulable volumetric model of the firstobject based on the captured depth data and the captured 2D video datafrom operation 1002. In certain examples, the individually-manipulablevolumetric model of the first object may be configured to beindividually manipulated with respect to an immersive virtual realityworld while a user of a media player device is experiencing theimmersive virtual reality world using the media player device. Forexample, the immersive virtual reality world may be based on virtualreality media content provided to the media player device andrepresentative of the immersive virtual reality world. Operation 1006may be performed in any of the ways described herein.

In operation 1008, the virtual reality media provider system may providevirtual reality media content representative of an immersive virtualreality world to a media player device associated with a user. Operation1008 may be performed in any of the ways described herein.

In operation 1010, the virtual reality media provider system mayindividually manipulate the individually-manipulable volumetric model ofthe first object generated in operation 1006 within the virtual realitymedia content provided in operation 1008. For example, the virtualreality media provider system may individually manipulate theindividually-manipulable volumetric model of the first object withrespect to the immersive virtual reality world represented in thevirtual reality media content. In some examples, the virtual realitymedia provider system may individually manipulate theindividually-manipulable volumetric model of the first object while theuser is experiencing the immersive virtual reality world use the mediaplayer device. Operation 1010 may be performed in any of the waysdescribed herein.

FIG. 11 illustrates an exemplary method 1100 for creating andmanipulating an individually-manipulable volumetric model of an objectlocated in a natural setting. While FIG. 11 illustrates exemplaryoperations according to one embodiment, other embodiments may omit, addto, reorder, and/or modify any of the operations shown in FIG. 11. Oneor more of the operations shown in FIG. 11 may be performed by system100 and/or any implementation thereof.

In operation 1102, a virtual reality media provider system that includesa configuration of synchronous video and depth capture devices disposedat fixed positions in a vicinity of a first person may capture 2D videodata and depth data for a surface of the first person. In some examples,the virtual reality media provider system may capture the 2D video dataand depth data in real time and the first person may be located in anatural setting along with one or more objects. Operation 1102 may beperformed in any of the ways described herein.

In operation 1104, the virtual reality media provider system maydistinguish the first person from the one or more objects located in thenatural setting along with the first person. For example, the virtualreality media provider system may distinguish the first person from theone or more objects based on the captured depth data and the captured 2Dvideo data captured in operation 1102. In some examples, operation 1104may be performed in real time. Operation 1104 may be performed in any ofthe ways described herein.

In operation 1106, the virtual reality media provider system maygenerate an individually-manipulable volumetric model of the firstperson. For example, operation 1106 may be performed based on thecaptured depth data and the captured 2D video data captured in operation1102. In some examples, operation 1106 may be performed in real time.Operation 1106 may be performed in any of the ways described herein.

In operation 1108, the virtual reality media provider system may providevirtual reality media content representative of an immersive virtualreality world to a media player device associated with a user. Forexample, the virtual reality media content may include theindividually-manipulable volumetric model of the first person generatedin operation 1106 performing an action within the immersive virtualreality world. Operation 1108 may be performed in any of the waysdescribed herein.

In operation 1110, the virtual reality media provider system may replacethe individually-manipulable volumetric model of the first person withan individually-manipulable volumetric model of a second person withinthe virtual reality media content provided in operation 1108. Forexample, the virtual reality media provider system may replace theindividually-manipulable volumetric model of the first person with theindividually-manipulable volumetric model of the second person such thatthe individually-manipulable volumetric model of the second personperforms the action performed by the individually-manipulable volumetricmodel of the first person in operation 1108 within the immersive virtualreality world while the user is experiencing the immersive virtualreality world using the media player device. In some examples, operation1110 may be performed in real time. Operation 1110 may be performed inany of the ways described herein.

In certain embodiments, one or more of the systems, components, and/orprocesses described herein may be implemented and/or performed by one ormore appropriately configured computing devices. To this end, one ormore of the systems and/or components described above may include or beimplemented by any computer hardware and/or computer-implementedinstructions (e.g., software) embodied on at least one non-transitorycomputer-readable medium configured to perform one or more of theprocesses described herein. In particular, system components may beimplemented on one physical computing device or may be implemented onmore than one physical computing device. Accordingly, system componentsmay include any number of computing devices, and may employ any of anumber of computer operating systems.

In certain embodiments, one or more of the processes described hereinmay be implemented at least in part as instructions embodied in anon-transitory computer-readable medium and executable by one or morecomputing devices. In general, a processor (e.g., a microprocessor)receives instructions, from a non-transitory computer-readable medium,(e.g., a memory, etc.), and executes those instructions, therebyperforming one or more processes, including one or more of the processesdescribed herein. Such instructions may be stored and/or transmittedusing any of a variety of known computer-readable media.

A computer-readable medium (also referred to as a processor-readablemedium) includes any non-transitory medium that participates inproviding data (e.g., instructions) that may be read by a computer(e.g., by a processor of a computer). Such a medium may take many forms,including, but not limited to, non-volatile media, and/or volatilemedia. Non-volatile media may include, for example, optical or magneticdisks and other persistent memory. Volatile media may include, forexample, dynamic random access memory (“DRAM”), which typicallyconstitutes a main memory. Common forms of computer-readable mediainclude, for example, a disk, hard disk, magnetic tape, any othermagnetic medium, a compact disc read-only memory (“CD-ROM”), a digitalvideo disc (“DVD”), any other optical medium, random access memory(“RAM”), programmable read-only memory (“PROM”), electrically erasableprogrammable read-only memory (“EPROM”), FLASH-EEPROM, any other memorychip or cartridge, or any other tangible medium from which a computercan read.

FIG. 12 illustrates an exemplary computing device 1200 that may bespecifically configured to perform one or more of the processesdescribed herein. As shown in FIG. 12, computing device 1200 may includea communication interface 1202, a processor 1204, a storage device 1206,and an input/output (“I/O”) module 1208 communicatively connected via acommunication infrastructure 1210. While an exemplary computing device1200 is shown in FIG. 12, the components illustrated in FIG. 12 are notintended to be limiting. Additional or alternative components may beused in other embodiments. Components of computing device 1200 shown inFIG. 12 will now be described in additional detail.

Communication interface 1202 may be configured to communicate with oneor more computing devices. Examples of communication interface 1202include, without limitation, a wired network interface (such as anetwork interface card), a wireless network interface (such as awireless network interface card), a modem, an audio/video connection,and any other suitable interface.

Processor 1204 generally represents any type or form of processing unitcapable of processing data or interpreting, executing, and/or directingexecution of one or more of the instructions, processes, and/oroperations described herein. Processor 1204 may direct execution ofoperations in accordance with one or more applications 1212 or othercomputer-executable instructions such as may be stored in storage device1206 or another computer-readable medium.

Storage device 1206 may include one or more data storage media, devices,or configurations and may employ any type, form, and combination of datastorage media and/or device. For example, storage device 1206 mayinclude, but is not limited to, a hard drive, network drive, flashdrive, magnetic disc, optical disc, RAM, dynamic RAM, other non-volatileand/or volatile data storage units, or a combination or sub-combinationthereof. Electronic data, including data described herein, may betemporarily and/or permanently stored in storage device 1206. Forexample, data representative of one or more executable applications 1212configured to direct processor 1204 to perform any of the operationsdescribed herein may be stored within storage device 1206. In someexamples, data may be arranged in one or more databases residing withinstorage device 1206.

I/O module 1208 may include one or more I/O modules configured toreceive user input and provide user output. One or more I/O modules maybe used to receive input for a single virtual reality experience. I/Omodule 1208 may include any hardware, firmware, software, or combinationthereof supportive of input and output capabilities. For example, I/Omodule 1208 may include hardware and/or software for capturing userinput, including, but not limited to, a keyboard or keypad, atouchscreen component (e.g., touchscreen display), a receiver (e.g., anRF or infrared receiver), motion sensors, and/or one or more inputbuttons.

I/O module 1208 may include one or more devices for presenting output toa user, including, but not limited to, a graphics engine, a display(e.g., a display screen), one or more output drivers (e.g., displaydrivers), one or more audio speakers, and one or more audio drivers. Incertain embodiments, I/O module 1208 is configured to provide graphicaldata to a display for presentation to a user. The graphical data may berepresentative of one or more graphical user interfaces and/or any othergraphical content as may serve a particular implementation.

In some examples, any of the facilities described herein may beimplemented by or within one or more components of computing device1200. For example, one or more applications 1212 residing within storagedevice 1206 may be configured to direct processor 1204 to perform one ormore processes or functions associated with data capture facility 102,data processing facility 104, or data distribution facility 106 ofsystem 100 (see FIG. 1). Likewise, storage facility 108 of system 100may be implemented by or within storage device 1206.

To the extent the aforementioned embodiments collect, store, and/oremploy personal information provided by individuals, it should beunderstood that such information shall be used in accordance with allapplicable laws concerning protection of personal information.Additionally, the collection, storage, and use of such information maybe subject to consent of the individual to such activity, for example,through well known “opt-in” or “opt-out” processes as may be appropriatefor the situation and type of information. Storage and use of personalinformation may be in an appropriately secure manner reflective of thetype of information, for example, through various encryption andanonymization techniques for particularly sensitive information.

In the preceding description, various exemplary embodiments have beendescribed with reference to the accompanying drawings. It will, however,be evident that various modifications and changes may be made thereto,and additional embodiments may be implemented, without departing fromthe scope of the invention as set forth in the claims that follow. Forexample, certain features of one embodiment described herein may becombined with or substituted for features of another embodimentdescribed herein. The description and drawings are accordingly to beregarded in an illustrative rather than a restrictive sense.

What is claimed is:
 1. A method comprising: receiving, by a virtualreality media provider system, two-dimensional (“2D”) video data anddepth data for surfaces of a dynamic object and a static object locatedin a natural setting, the 2D video data and depth data captured by afirst capture device and a second capture device disposed at differentpositions with respect to the dynamic and static objects;distinguishing, by the virtual reality media provider system, thedynamic object from the static object by performing a plurality oftechniques, the plurality of techniques including: determining, based onthe received 2D video data, that the dynamic object is moving inrelation to the static object, and determining, based on the received 2Dvideo data, that, from a vantage point of at least one of the differentpositions at which the first and second capture devices are disposed, arepresentation of the dynamic object captured within the 2D video datadoes not overlap with a representation of the static object; andgenerating and updating, by the virtual reality media provider systembased on the received 2D video data and depth data and thedistinguishing of the dynamic object from the static object, anindividually-manipulable volumetric model of the dynamic object and anindividually-manipulable volumetric model of the static object; wherein:the generating of the individually-manipulable volumetric model of thedynamic object includes capturing a behavior of the dynamic object andstoring the captured behavior with the individually-manipulablevolumetric model of the dynamic object for use in making anindividually-manipulable volumetric model of an additional dynamicobject that replaces the individually-manipulable volumetric model ofthe dynamic object perform the behavior of the dynamic object, and theupdating of the individually-manipulable volumetric model of the dynamicobject is performed more regularly than the updating of theindividually-manipulable volumetric model of the static object.
 2. Themethod of claim 1, wherein the first and second capture devices areimplemented as synchronous video and depth capture devices.
 3. Themethod of claim 1, wherein the plurality of techniques further includesdetermining, based on the received depth data, that the dynamic objectis located at least a predetermined distance away from the staticobject.
 4. The method of claim 1, wherein the plurality of techniquesfurther includes: identifying, based on the received 2D video data, thedynamic object and the static object; and determining, based on theidentifying of the dynamic and static objects, that the dynamic objectis different from the static object.
 5. The method of claim 1, wherein:the natural setting is a setting of a real-world event; the differentpositions with respect to the dynamic and static objects where the firstand second capture devices are disposed include a plurality of fixedpositions surrounding the dynamic and static objects within the settingof the real-world event.
 6. The method of claim 1, wherein: theindividually-manipulable volumetric model of the dynamic object isconfigured to be individually manipulated with respect to an immersivevirtual reality world while a user of a media player device isexperiencing the immersive virtual reality world using the media playerdevice; and the immersive virtual reality world is based on virtualreality media content provided to the media player device andrepresentative of the immersive virtual reality world.
 7. The method ofclaim 6, wherein the individually-manipulable volumetric model of thedynamic object is configured to be individually manipulated with respectto the immersive virtual reality world by inserting theindividually-manipulable volumetric model of the dynamic object into theimmersive virtual reality world at a particular location within theimmersive virtual reality world.
 8. The method of claim 6, wherein theindividually-manipulable volumetric model of the dynamic object isconfigured to be individually manipulated with respect to the immersivevirtual reality world by: replacing the individually-manipulablevolumetric model of the dynamic object included within the immersivevirtual reality world with the individually-manipulable volumetric modelof the additional dynamic object within the immersive virtual realityworld; and making the individually-manipulable volumetric model of theadditional dynamic object perform the behavior of the dynamic objectbased on the captured behavior stored with the individually-manipulablevolumetric model of the dynamic object.
 9. The method of claim 6,wherein the individually-manipulable volumetric model of the dynamicobject is configured to be individually manipulated with respect to theimmersive virtual reality world by removing the individually-manipulablevolumetric model of the dynamic object from the immersive virtualreality world.
 10. A system comprising: a first capture device disposedat a first position with respect to a dynamic object and a static objectlocated in a natural setting, the first capture device configured tocapture two-dimensional (“2D”) video data and depth data for surfaces ofthe dynamic and static objects from the first position; a second capturedevice disposed at a second position with respect to the dynamic andstatic objects, the second capture device configured to further capturethe 2D video data and depth data for the surfaces of the dynamic andstatic objects from the second position; a memory storing instructions;and a processor communicatively coupled to the memory and to the firstand second capture devices, the processor configured to execute theinstructions to: receive, from the first and second capture devices, the2D video data and depth data for the surfaces of the dynamic and staticobjects; distinguish the dynamic object from the static object byperforming a plurality of techniques, the plurality of techniquesincluding: determining, based on the received 2D video data, that thedynamic object is moving in relation to the static object, anddetermining, based on the received 2D video data, that, from a vantagepoint of at least one of the first and second positions, arepresentation of the dynamic object captured within the 2D video datadoes not overlap with a representation of the static object; andgenerate and update, based on the received 2D video data and depth dataand the distinguishing of the dynamic object from the static object, anindividually-manipulable volumetric model of the dynamic object and anindividually-manipulable volumetric model of the static object; wherein:the generating of the individually-manipulable volumetric model of thedynamic object includes capturing a behavior of the dynamic object andstoring the captured behavior with the individually-manipulablevolumetric model of the dynamic object for use in making anindividually-manipulable volumetric model of an additional dynamicobject that replaces the individually-manipulable volumetric model ofthe dynamic object perform the behavior of the dynamic object, and theupdating of the individually-manipulable volumetric model of the dynamicobject is performed more regularly than the updating of theindividually-manipulable volumetric model of the static object.
 11. Thesystem of claim 10, wherein the first and second capture devices areimplemented as synchronous video and depth capture devices.
 12. Thesystem of claim 10, wherein the plurality of techniques furtherincludes: determining, based on the received depth data, that thedynamic object is located at least a predetermined distance away fromthe static object; and identifying, based on the received 2D video data,the dynamic object and the static object and determining that thedynamic object is different from the static object.
 13. The system ofclaim 10, wherein: the natural setting is a setting of a real-worldevent; the first and second positions with respect to the dynamic andstatic objects where the first and second capture devices arerespectively disposed are fixed positions surrounding the dynamic andstatic objects within the setting of the real-world event.
 14. Thesystem of claim 10, wherein: the individually-manipulable volumetricmodel of the dynamic object is configured to be individually manipulatedwith respect to an immersive virtual reality world while a user of amedia player device is experiencing the immersive virtual reality worldusing the media player device; and the immersive virtual reality worldis based on virtual reality media content provided to the media playerdevice and representative of the immersive virtual reality world. 15.The system of claim 14, wherein the individually-manipulable volumetricmodel of the dynamic object is configured to be individually manipulatedwith respect to the immersive virtual reality world by at least one of:inserting the individually-manipulable volumetric model of the dynamicobject into the immersive virtual reality world at a particular locationwithin the immersive virtual reality world; replacing theindividually-manipulable volumetric model of the dynamic object includedwithin the immersive virtual reality world with theindividually-manipulable volumetric model of the additional dynamicobject within the immersive virtual reality world and making theindividually-manipulable volumetric model of the additional dynamicobject perform the behavior of the dynamic object based on the capturedbehavior stored with the individually-manipulable volumetric model ofthe dynamic object; and removing the individually-manipulable volumetricmodel of the dynamic object from the immersive virtual reality world.16. A non-transitory computer-readable medium storing instructions that,when executed, direct a processor of a computing device to: receivetwo-dimensional (“2D”) video data for surfaces of a dynamic object and astatic object located in a natural setting, the 2D video data and depthdata captured by a first capture device and a second capture devicedisposed at different positions with respect to the dynamic and staticobjects; distinguish the dynamic object from the static object byperforming a plurality of techniques, the plurality of techniquesincluding: determining, based on the received 2D video data, that thedynamic object is moving in relation to the static object, anddetermining, based on the received 2D video data, that, from a vantagepoint of at least one of the different positions at which the first andsecond capture devices are disposed, a representation of the dynamicobject captured within the 2D video data does not overlap with arepresentation of the static object; and generate and update, based onthe received 2D video data and depth data and the distinguishing of thedynamic object from the static object, an individually-manipulablevolumetric model of the dynamic object and an individually-manipulablevolumetric model of the static object; wherein: the generating of theindividually-manipulable volumetric model of the dynamic object includescapturing a behavior of the dynamic object and storing the capturedbehavior with the individually-manipulable volumetric model of thedynamic object for use in making an individually-manipulable volumetricmodel of an additional dynamic object that replaces theindividually-manipulable volumetric model of the dynamic object performthe behavior of the dynamic object, and the updating of theindividually-manipulable volumetric model of the dynamic object isperformed more regularly than the updating of theindividually-manipulable volumetric model of the static object.
 17. Thenon-transitory computer-readable medium of claim 16, wherein the firstand second capture devices are implemented as synchronous video anddepth capture devices.
 18. The non-transitory computer-readable mediumof claim 17, wherein the plurality of techniques further includes:determining, based on the received depth data, that the dynamic objectis located at least a predetermined distance away from the staticobject; and identifying, based on the received 2D video data, thedynamic object and the static object and thereby determining that thedynamic object is different from the static object.
 19. Thenon-transitory computer-readable medium of claim 16, wherein: thenatural setting is a setting of a real-world event; the differentpositions with respect to the dynamic and static objects where the firstand second capture devices are disposed include a plurality of fixedpositions surrounding the dynamic and static objects within the settingof the real-world event.
 20. The non-transitory computer-readable mediumof claim 16, wherein: the individually-manipulable volumetric model ofthe dynamic object is configured to be individually manipulated withrespect to an immersive virtual reality world while a user of a mediaplayer device is experiencing the immersive virtual reality world usingthe media player device; and the immersive virtual reality world isbased on virtual reality media content provided to the media playerdevice and representative of the immersive virtual reality world.